MDEV-39509: Fix "invalid iv length" with OpenSSL 4.0
authorGeorg Richter <georg@mariadb.com>
Sat, 2 May 2026 09:44:03 +0000 (11:44 +0200)
committerOtto Kekäläinen <otto@debian.org>
Tue, 2 Jun 2026 00:02:02 +0000 (00:02 +0000)
commit9734b1d6308f724c1ef8cd756af37f6bb3a66df5
treec7b41d0c0626e7b4d2fbf4a2aee1353c4754ec2a
parent46802c20d0052babc8069d4dc3d6f472ecf2310d
MDEV-39509: Fix "invalid iv length" with OpenSSL 4.0

OpenSSL 4.0 changed the behavior of EVP_CIPHER_CTX_new:
It now initializes the internal iv_len with a sentinel
value of -1. When calling EVP_CipherInit_ex with the
cipher, key, and IV simultaneously, the provider validates
the IV against this sentinel before the cipher's default
metadata is applied, triggering an error.

This patch implements a two-stage initialization for OpenSSL 4.0+:
- Call EVP_CipherInit_ex with only the cipher to "thaw" the
  context and set the correct default iv_len.
- Perform a second call to provide the actual Key and IV data.

This approach ensures compatibility with OpenSSL 4.0 while
avoiding EVP_CipherInit_ex2 to maintain support for FIPS mode
and older versions.

(cherry picked from commit 5ca4ab639ad5f99d74f794b54ba1b4d6182ec2a9 from
upstream feature branch, not yet merged)

Forwarded: https://github.com/MariaDB/server/pull/5029

Gbp-Pq: Name MDEV-39509-Fix-invalid-iv-length-with-OpenSSL-4.0.patch
mysys_ssl/my_crypt.cc